Unsupervised learning of paraphrase/translation alternations and selective application thereof

ABSTRACT

A system is disclosed for selectively applying a paraphrase alternation pattern to a textual input. The system includes a pattern identification component for processing a collection of data and identifying a plurality of potentially applicable paraphrase alternation patterns. A context model provides an objective frame of reference in which to compare one or more of the plurality of potentially applicable paraphrase alternation patterns to the textual input to determine whether the pattern can be applied without changing meaning. A language model provides a principled basis for determining the boundaries of the text segment to be modified.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to application Ser. No. 10/988,721,filed Nov. 15, 2004, and application Ser. No. 10/989,076, filed Nov. 15,2004, identified by the contents of which are hereby incorporated byreference in their entireties.

BACKGROUND OF THE INVENTION

The present invention deals with knowledge-poor sub-sententialparaphrasing. More specifically, the present invention deals withmethods for learning meaning-preserving text segment alternations fromword-aligned, parallel text (either monolingual or bilingual). Thepresent invention also deals with selectively applying such alternationswithout introducing ambiguity and/or changing meaning.

The recognition and generation of paraphrases is a key problem for manyapplications of Natural Language Processing (NLP) systems. Being able toidentify that two different pieces of text are equivalent in meaningenables a system to behave much more intelligently. A fundamental goalof work in this area is to produce a program that will be able tore-state a text segment in a manner that preserves its semantic contentwhile manipulating features like vocabulary, word order, reading level,and degree of conciseness or verbosity.

One exemplary application which can benefit from paraphraseidentification and generation includes a question answering system. Forexample, consider a question “When did the Governor of California arrivein Sacramento?”. It is very likely that a large data corpus, such as aglobal computer network (or a news reporting system that publishesarticles on a global computer network) may already contain text thatanswers the question. In fact, such a corpus may already contain textthat answers the question and is phrased in exactly the same terms asthe question. Therefore, a conventional search engine may have nodifficulty in finding text that matches the question, and thus returningan adequate result.

The same problem becomes more difficult when searching a smaller datacorpus, such as one found on an intranet. In that case, even though thesmall data corpus may contain text that answers the question, the answermay be phrased in different terms than the question. By way of example,the following sentence answers the question set out above, but isphrased in different terms from the question:

-   -   The California Governor landed in Sacramento on Sep. 20, 2004.

Since this answer is phrased differently than the question, aconventional search engine may encounter difficulty in returning a goodresult, given only the described textual answer in the corpus that it issearching.

Prior systems for addressing the problem of recognition and generationof paraphrases include large hand-coded efforts that attempt to addressthe problem in limited contexts. For example, large hand-coded systemsattempt to map between a wide variety of different ways of saying thesame thing and a form acceptable to a command and control system. Ofcourse, this is extremely difficult because the author of the codelikely cannot think of every different way a user might phrasesomething. Therefore, the focus in the research community has shiftedfrom manual efforts to automatic methods of paraphrase identificationand generation.

Recent work on systems aimed at automatically identifying textualparaphrase relations includes D. Lin and P. Pantel, DIRT-DISCOVERY OFINFERENCE RULES FROM TEXT, Proceedings of ACMSIGKDD Conference onKnowledge Discovery and Data Mining, pages 323-328 (2001). The DIRTarticle examines the distributional properties of dependency pathslinking identical “anchor points” (i.e. identical or similar words) in aparsed corpus of newswire data. None of the special properties of newsdata are exploited since the parsed corpus is simply viewed as a largesource of monolingual data. The basic idea is that high frequencydependency graph paths which link identical or similar words arethemselves likely to be similar in meaning. When run over a gigabyte ofnewspaper data, the system identified patterns such as:

-   -   X is resolved by Y.    -   X resolves Y.    -   X finds a solution to Y.    -   X tries to solve Y.

The DIRT system has been limited to a very restricted sort of “triple”relation, such as “X verb Y”.

Another article that deals with paraphrase identification is Y.Shinyama, S. Sekine, K. Sudo and R. Grisham, AUTOMATIC PARAPHRASEACQUISITION FROM NEWS ARTICLES, Proceedings of Human Language TechnologyConference, San Diego, Calif. (HLT 2002). In the Shinyama et al.article, the observation is made that articles from different newspapersthat describe the same event often exemplify paraphrase relations. Thepaper describes a technique that relies on the assumption that namedentities (such as people, places, dates and addresses) remain constantacross different newspaper articles on the same topic or on the sameday. Articles are clustered using an existing information retrievalsystem into, for example, “murder” or “personnel” groupings or clusters.Named entities are annotated using a statistical tagger, and the data isthen subjected to morphological and syntactic analysis to producesyntactic dependency trees. Within each cluster, sentences are clusteredbased on the named entities they contain. For instance, the followingsentences are clustered because they share the same four named entities:

-   -   Vice President Osamu Kuroda of Nihon Yamamuri Glass Corp. was        promoted to President.    -   Nihon Yamamuri Glass Corp. decided the promotion of Vice        President Osamu Kuroda to President on Monday.

Given the overlap in named entities, these sentences are assumed to belinked by a paraphrase relationship. Shinyama et al. then attempt toidentify patterns that link these sentences using existing machineryfrom the field of information extraction.

Shinyama et al. also attempt to learn very simple phrase level patterns,but the technique is limited by its reliance on named entity anchorpoints. Without these easily identified anchors, Shinyama et al. canlearn nothing from a pair of sentences. The patterns that Shinyama etal. learn all center on the relationship between a particular type ofentity and some type of event within a particular domain. The resultsare fairly poor, particularly when the training sentences contain veryfew named entities.

Another article also deals with paraphrases. In Barzilay R. and L. Lee,LEARNING TO PARAPHRASE: AN UNSUPERVISED APPROACH USING MULTIPLE-SEQUENCEALIGNMENT, Proceedings of HLT/NAACL: (2003), Edmonton, Canada, topicdetection software is used to cluster thematically similar newspaperarticles from a single source, and from several years worth of data.More specifically, Barzilay et al. attempts to identify articlesdescribing terrorist incidents. They then cluster sentences from thesearticles in order to find sentences that share a basic overall form orthat share multiple key words. These clusters are used as the basis forbuilding templatic models of sentences that allow for certainsubstitutional elements. In short, Barzilay et al. focuses on findingsimilar descriptions of different events, even events which may haveoccurred years apart. This focus on grouping sentences by form meansthat this technique will not find some of the more interestingparaphrases.

Also Barzilay and Lee require a strong word order similarity in order toclass two sentences as similar. For instance, they may not class evenactive/passive variants of an event description as related. Thetemplatic paraphrase relationships learned by Barzilay et al. arederived from sets of sentences that share an overall fixed word order.The paraphrases learned by the system amount to regions of flexibilitywithin this larger fixed structure. It should also be noted thatBarzilay and Lee appear to be alone in the literature in proposing ageneration scheme. The other work discussed in this section is aimedonly at recognizing paraphrases.

Another paper, Barzilay and McKeown Extracting Paraphrases From aParallel Corpus, Proceedings of ACL/EACL (2001), relies on multipletranslations of a single source document. However, Barzilay and McKeownspecifically distinguish their work from machine translation techniques.They state that without a complete match between words in relatedsentences, one is prevented from using “methods developed in the MTcommunity based on clean parallel corpora.” Thus, Barzilay and McKeownreject the idea that standard machine translation techniques could beapplied to the task of learning monolingual paraphrases.

Another prior art system also deals with paraphrases. This system relieson multiple translations of a single source to build finite staterepresentations of paraphrase relationships. B. Pang, K. Knight, and D.Marcu, SYNTAX BASED ALIGNMENT OF MULTIPLE TRANSLATION: EXTRACTINGPARAPHRASES AND GENERATING NEW SENTENCES, Proceedings of NAACL-HLT,2003.

Still another prior reference also deals with paraphrase recognition.Ibrahim, Ali, EXTRACTING PARAPHRASES FROM ALIGNED CORPORA, MasterThesis, MIT (2002). In his thesis, Ibrahim indicates that sentences are“aligned” or subjected to “alignment” and that paraphrases areidentified. However, the term “alignment” as used in the thesis meanssentence alignment instead of word or phrase alignment and does notrefer to the conventional word and phrase alignment performed in machinetranslation systems. Instead, the alignment discussed in the thesis isbased on the following paper, which attempts to align sentences in onelanguage to their corresponding translations in another:

Gale, William, A. and Church, Kenneth W., A PROGRAM FOR ALIGNINGSENTENCES IN BILINGUAL CORPORA, Proceedings of the Associations forComputational Linguistics, Pages 177-184 (1991). Ibrahim uses thisalgorithm to align sentences within multiple English translations of,for example, Jules Verne novels. However, sentence structure can varydramatically from translation to translation. What one translatorrepresents as a single long sentence, another might map to two shorterones. This means that the overall number of sentences in the differenttranslations of a single novel do not match, and some sort of automatedsentence alignment procedure is needed to identify equivalent sentences.The overall technique Ibrahim uses for extracting paraphrases from thesealigned monolingual sentences is derived from the multiple-translationconcepts set forth in the Barzilay, McKeown reference, plus a variationon the DIRT framework described by Lin et al.

SUMMARY OF THE INVENTION

Embodiments of the present invention pertain to a system for selectivelyapplying a paraphrase alternation pattern to a textual input. The systemincludes a pattern identification component for processing a collectionof data and identifying a plurality of potentially applicable paraphrasealternation patterns. A context model provides an objective frame ofreference in which to compare one or more of the plurality ofpotentially applicable paraphrase alternation patterns to the textualinput to determine whether the pattern can be applied without changingmeaning. A language model provides a principled basis for determiningthe boundaries of the text segment to be modified.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of one embodiment of an environment in whichthe present invention can be used.

FIG. 2 is a block diagram of a paraphrase recognition and generationsystem in accordance with one embodiment of the present invention.

FIG. 3 is a flow chart illustrating the operation of the system shown inFIG. 2.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The present invention deals with methods for learning meaning-preservingtext segment alternations from word-aligned, parallel text (eithermonolingual or bilingual). It should be noted that the term “textsegment,” as it is used herein, is intended to refer to any contiguouschunk of text, from an entire document down to a single word. Thepresent invention also deals with selectively applying such alternationsso as to preserve meaning and/or avoid ambiguity. However, prior todiscussing the present invention in greater detail, one illustrativeenvironment in which the present invention can be used will bediscussed.

FIG. 1 illustrates an example of a suitable computing system environment100 on which the invention may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing environment100 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplaryoperating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both locale and remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a locale bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) locale bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computer 100. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier WAVor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, FR,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way o example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removablevolatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies.

A user may enter commands and information into the computer 110 throughinput devices such as a keyboard 162, a microphone 163, and a pointingdevice 161, such as a mouse, trackball or touch pad. Other input devices(not shown) may include a joystick, game pad, satellite dish, scanner,or the like. These and other input devices are often connected to theprocessing unit 120 through a user input interface 160 that is coupledto the system bus, but may be connected by other interface and busstructures, such as a parallel port, game port or a universal serial bus(USB). A monitor 191 or other type of display device is also connectedto the system bus 121 via an interface, such as a video interface 190.In addition to the monitor, computers may also include other peripheraloutput devices such as speakers 197 and printer 196, which may beconnected through an output peripheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a hand-helddevice, a server, a router, a network PC, a peer device or other commonnetwork node, and typically includes many or all of the elementsdescribed above relative to the computer 110. The logical connectionsdepicted in FIG. 1 include a locale area network (LAN) 171 and a widearea network (WAN) 173, but may also include other networks. Suchnetworking environments are commonplace in offices, enterprise-widecomputer networks, intranets and the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user-inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on remote computer 180. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

It should be noted that the present invention can be carried out on acomputer system such as that described with respect to FIG. 1. However,the present invention can be carried out on a server, a computer devotedto message handling, or on a distributed system in which differentportions of the present invention are carried out on different parts ofthe distributed computing system.

FIG. 2 is a block diagram of one embodiment of a paraphrase processingsystem 200. System 200 has access to a document database 202 andincludes a document clustering system 204, text segment selection system206, word/phrase alignment system 210, pattern identifier 242, contextmodel 246, language model 248 and paraphrase generator 249. FIG. 3 is aflow diagram illustrating the operation of system 200 in accordance withone aspect of the present invention.

Document database 202 illustratively includes a variety of differentnews articles written by a variety of different news agencies. Each ofthe articles illustratively includes a time stamp indicatingapproximately when the article was authored. Also, the plurality ofarticles from the different news agencies will illustratively be writtenabout a wide variety of different events.

Of course, while the present invention is described with respect to newsarticles, other source documents could be used as well, such astechnical articles describing a common process, different medicalarticles describing a common medical procedure, etc.

Document clustering system 204 accesses document database 202 asillustrated by block 214 in FIG. 3. It should also be noted that while asingle database 202 is illustrated in FIG. 2, a plurality of databasescould be accessed instead.

Clustering system 204 identifies articles in document database 202 thatare written about the same event. In one embodiment, the articles arealso identified as being written at approximately the same time (such aswithin a predetermined time threshold of one another, e.g., one month,one week, one day, within hours, etc. . . . as desired). The articlesidentified as being written about the same event (and perhaps at aboutthe same time) form a document cluster 218. This is indicated by block216 in FIG. 3.

Once related source articles are identified as a cluster 218, desiredtext segments (such as sentences, phrases, headlines, paragraphs, etc.)in those articles are extracted. For instance, a journalistic conventionin news articles advises that the first 1-2 sentences of the articlerepresent a summary of the rest of the article. Therefore, in accordancewith one embodiment of the present invention, the articles (which wereillustratively written by different news agencies) are clustered intoclusters 218 and provided to text segment selection system 206 where thefirst two sentences of each article, in each cluster 218, are extracted.While the present discussion proceeds with respect to sentences, it willbe noted that this is exemplary only and other text segments could justas easily be used. The sentences from each cluster 218 of articles areoutput as a sentence set 222 corresponding to the clustered articles.The sentence sets 222 are output by text segment selection system 206 toword/phrase alignment system 210. This is indicated by block 220 in FIG.3.

In the specific example in which sentences are used, many of thesentences gathered in this way appear to be versions of some singleoriginal source sentence, slightly rewritten by editors at differentnews agencies for stylistic reasons. Frequently, these sets of sentenceshave been observed to differ only in small ways, such as the order ofthe clauses appearing in the sentence.

Text segment selection system 206 generates sets 222 of sentences foreach cluster. It should be noted that the word/phrase alignment system210 can operate on large sets of sentences by extracting mappingsbetween words or phrases based on a holistic examination of thesentences in the set. However, the present discussion proceeds withrespect to generating sentence pairs and performing alignment on thosepairs, as but one illustrative embodiment. Thus, in one embodiment, theidentified sets of sentences are formed into pairs of sentences.Therefore, text segment selection system 206 pairs each sentence in aset against every other sentence in that set to generate sentence pairsfor each set. The sentence pairs are in one embodiment, subjected to anoptional filtering step, and in another embodiment, are output directlyto word/phrase alignment system 210. While the filtering will bedescribed with respect to the present embodiment, it will be noted thatthe steps associated with filtering are optional.

In one illustrative embodiment, text segment selection system 206implements a heuristic that filters the sentence pairs based on sharedkey content words. For example, in one illustrative embodiment, system206 filters the sentence pairs, removing those sentence pairs that donot share at least three words of at least four characters each. Ofcourse, filtering is optional, and, if used, the filtering algorithmimplemented can vary widely. Any of a variety of different filteringtechniques can be used, such as filtering on past results (whichrequires a feedback loop in the output of word/phrase alignment system210 back to text segment selection system 206), filtering on differentnumbers of content words, filtering on other semantic or syntacticinformation, etc. In any case, the sets of sentences can be paired andcan be filtered and provided to word/phrase alignment system 210.

In one embodiment, alignment system 210 develops alignment models and/orperforms statistical word and/or phrase alignment on sentence sets 222.This is indicated by block 230 in FIG. 3. The output of aligned wordsand/or phrases is indicated by block 240 in FIG. 2 and step 236 in FIG.3.

In one illustrative embodiment, the word/phrase alignment system 210implements a conventional word/phrase alignment algorithm from thestatistical machine translation literature in an attempt to learnlexical correspondences between the sentences in the sets 222. In oneembodiment, some alignments will involve shifts of identical strings.For example:

-   1. . . . A of B . . . (Governor of California)-   1. . . . BA . . . (California Governor)    -   NOTE: of=“pivot”    -   NOTE: Pattern is bi-directional-   2. . . . A of the B . . . (President of the United States)-   2. . . . BA . . . (United States President)    -   NOTE: of the=“pivot phrase”    -   NOTE: Pattern is bi-directional

In another embodiment, alignments are not restricted to string-identicalscenarios. For example:

-   1. . . . the American ambassador . . .-   1. . . . the Ambassador of the United States . . .    Processing components known in the art can be leveraged to identify    alignments where phrases or words are not identical but still mapped    to one another. In one embodiment, known knowledge-poor word    alignment techniques are implemented to align text segments that are    likely to demonstrate similar or identical meanings.

In one embodiment, some alignments will involve alternations frompassive voice phrasing to active voice phrasing. For example,

-   1. The ball was hit by the boy.-   1. The boy hit the ball.

In accordance with one embodiment, alignments are mapped from onelanguage to another utilizing processing tools common to statisticalmachine translation technology. In another embodiment, tools forfacilitating alignment techniques are known within statistical machinetranslation technology and can be leveraged to accomplish alignment evenwithin a single language.

A more specific explanation of how statistical machine translationtechnology can be leveraged for meaning-oriented alignment will now bediscussed. It is standard practice in statistical machine translation toutilize a large corpus of paired sentences that are believed todemonstrate the same meaning in two languages. For example, a collectionof human-translated data can be utilized to train a statistical machinetranslation system. A translation lexicon is learned from this data inorder to capture translation patterns that have been identified usingstatistical techniques (e.g., this English word or phrase likelycorresponds to this Japanese or Spanish word or phrase). Thesetechniques permit the matching of character strings that are notidentical at string level, but which appear in analogous contexts sofrequently that it is likely that they mean the same thing (e.g.,English “dog”=Spanish “perro”). Translation lexicon learning techniquescan therefore be leveraged to support the identification of paraphrasealignments, such that different English text segments with same meaningswill map to the same outcome even though the character strings aredifferent.

In one illustrative embodiment, word/phrase alignment system 210 isimplemented using techniques set out in P. F. Brown et al., TheMathematics of Statistical Machine Translation: Parameter Estimation,Computational Linguistics, 19:263-312, (June 1993). Of course, othermachine translation or word/phrase alignment techniques can be used foridentifying associations between words and the input text. Any similarsystem could be implemented for similar purposes without departing fromthe scope of the present invention. Other applicable techniques aredescribed in Quirk, C., C. Brockett, and W. B. Dolan, MonolingualTranslation for Paraphrase Generation, In Proceedings of the 2004Conference on Empirical Methods in Natural Language Processing, 25-26Jul. 2004, Barcelona Spain, pp. 142-149, as well as in Dolan W. B., C.Quirk, and C. Brockett, Unsupervised Construction of Large ParaphraseCorpora: Exploiting Massively Parallel News Sources, Proceedings ofCOLING 2004, Geneva, Switzerland.

Thus, parallel collections of text are extracted from clustered newsstories. For example, the following pair of sentences are illustrativelyclustered together:

-   1. “Kerry challenged Bush to get out of the way”-   2. “Kerry said President Bush stands in the way”

Some identified parallel sentences will provide relatively small-scalestylistic alternations, for example:

-   1. The medicine called factor eight . . .-   1. The medicine—called factor viii . . .-   2. They are getting a tad more colorful . . .-   2. They are getting a tad more colourful . . .-   3. President and Chief Operating Officer-   3. President and COO . . .

Other collections reflect more complex meaning preserving syntacticalternations, such as:

-   1. . . . the plot to kidnap Victoria Beckham . . .-   1. . . . the Victoria Beckham kidnap case . . .-   2. . . . children's hospital at Montefiore . . .-   2. . . . Montefiore children's hospital . . .-   3. . . . 17-year-old Max Guarino . . .-   3. . . . Max Guarino, 17, . . .-   4. . . . said in a statement today . . .-   4. . . . said today in a statement . . .

In accordance with one aspect of the present invention, output 240 fromcomponent 210 is utilized as a basis for an unsupervised knowledge-poorprocess for identifying and learning textual alternations that aremeaning-preserving. These alternations can then be utilized as a basisfor identifying and generating meaning-preserving paraphrases in any ofa variety of contexts and applications. In the process, in oneembodiment, alternations that do not preserve meaning (e.g., “WashingtonUniversity” does not mean the same as “University of Washington”) can beidentified, learned and utilized as a basis for avoiding inaccuratealternations.

It should be noted that the term “knowledge-poor” as it is utilizedherein is generally intended to signify a lack of information aboutgrammatical structure, linguistic structure, dictionary information,syntactic structure, synonym information, parts of speech information,background information (e.g., California is a state, Bush is thePresident, etc.) or other similar information. Generally speaking, thegoal is to be applicable to support applications where access toextensive resources is not available, as may be the case when applyingthis method to languages other than English, and to provide a dynamicsystem that is not dependent upon a constant process of updating suchresources. Accordingly, one aspect of the present invention pertains toa knowledge-poor process for analyzing alternation/transformation-richdata to deduce meaning-preserving paraphrase alternations. This beingsaid, incorporation of information that are not consistent withknowledge-poor processing is also within the scope of the presentinvention.

In accordance with one aspect of the present invention, as is indicatedby block 250 in FIG. 3, the next step in the process is to utilizepattern identifier 242 to analyze output 240 and identify potentialparaphrase patterns. From each alignment, one or more alternationpatterns are illustratively derived.

In one embodiment, some patterns will contain pivots (e.g., words orphrases) and others will not. For example:

-   1. AB=BA (No pivot)    -   . . . said Rumsfield . . .    -   . . . Rumsfield said . . .    -   . . . was automatically shut down . . .    -   . . . shut down automatically . . .-   2. A, B=BA (With comma pivot)    -   . . . Steve Jobs, Apple CEO . . .    -   . . . Apple CEO Steve Jobs    -   . . . In a statement, @ Stake said . . .    -   . . . The @ Stake statement said . . .

Other potential patterns include but are certainly not limited to:

-   -   A in B=BA    -   A and B=B and A    -   A, B=B, A    -   A the B and BA    -   A were B=BA    -   A was B=BA    -   A's B=BA    -   A on B=BA    -   A's B=B of A        The alternation patterns shown above are given in English for        illustrative purposes only. Persons skilled in the art will        recognize that analogous alternation patterns can generally be        identified in any other languages using the method described in        this patent.

In accordance with one aspect of the present invention, as is indicatedin block 250 of FIG. 3, the identified patterns are sorted based onfrequency of occurrence. Patterns that occur in the data frequentlyreceive a high ranking while patterns that do not occur frequentlyreceive a low ranking. Accordingly, patterns having little value tosubsequent paraphrase processing will sort themselves out to the bottomof the list while more valuable patterns will rise to the top. Inaccordance with one embodiment, a cut-off point is selected above whichare patterns to be used for subsequent processing and below which arepatterns to be discounted. Longer and more complex meaning-preservingpatterns will eventually rise above the threshold as the quantity ofdata underlying the frequency-based list grows larger.

The described frequency sorting process is based on the assumption thathigh frequency alternation patterns are more likely to be meaningpreserving. Thus, given a sufficient quantity of training data, arelatively low-frequency meaning-preserving alternation pattern such asthe following is likely to eventually surface above the threshold:

-   1. A is the birthplace of B-   1. B was born in A    On the other hand, an alternation such as the following is    essentially accidental and will pull to the bottom of a    frequency-based list.-   1. U.S. President George Bush yesterday declared-   1. Bob declared that U.S. President George Bush

In accordance with block 252 in FIG. 3, the next step in the process isto process a textual input (244 in FIG. 2) to determine if the textualcomponents thereof are applicable to any of the patterns above thethreshold. Applicability of a pattern means there is a potential, butnot a guarantee, for a meaning-preserving alternation. In oneembodiment, before an alternation is proposed, a determination is madeas to whether the potential alternation introduces ambiguity and/oralters meaning.

In accordance with one aspect of the present invention, block 254represents applying a context model 246 (FIG. 2) to facilitate adetermination as to whether an applicable pattern introduces ambiguityand/or alters meaning. For example:

-   1. owner of Mets=>Mets owner    -   Pattern can apply-   2. because of drought=>drought because    -   Do not apply (meaning not preserved)-   3. cup of coffee=>coffee cup    -   Do not apply pattern (ambiguous result)-   4. hair of the dog=>dog hair    -   Do not apply (meaning not preserved)-   5. tooth enamel=>enamel of tooth    -   Do not apply (awkward result linguistically)

It should be pointed out that the patterns above the threshold line arealready biased toward meaning-preserving changes. However, applicationof context model 246 assists in determining when to apply a pattern toreal input data.

The process up to this point essentially guarantees that for eachgenerated pattern above the threshold there exists correspondingparallel sentences that represent the source of the pattern. All ofthese sentences are positive examples of contexts wherein the patterncan be executed in a meaning-preserving manner. In accordance with oneembodiment, application of the context model 246 in accordance withblock 254 involves training a statistical classifier on the sourcesentences to assist in the identification of other contexts in which thepattern can be applied successfully (i.e., without introducing ambiguityand/or altering meaning). Accordingly, the context model is utilized asa basis for determining whether a given pattern that matches the inputon the surface can be applied in a meaning-preserving manner.

Accordingly, in one aspect of the present invention, a context model isgenerated for each pattern above the threshold in order to identifyspecific contexts in which the pattern can be applied. The context modelis derived based on the sets of information utilized as a basis forgenerating the pattern. Generally speaking, the context model is ageneralization of basic patterns that define other circumstances underwhich the pattern can be applied in a meaning-preserving manner.

There are many specific bases for generation of a context model withinthe scope of the present invention. Any basis, whether complex orsimple, that supports generation of a suitable predictive inference tobe generated is within the scope of the present invention. Such aninference will be a generalized assumption that a contextualcharacteristic supported in the data underlying the pattern is reflectedin the context of the input 244 so as to justify a conclusion that asimilar alternation will also be meaning-preserving. In accordance withone aspect of the present invention, the basis of the context modelapplication supports the overall goal of preserving “knowledge-poor”processing. This being said, bases for other context model applicationsthat are not consistent with knowledge-poor processing are also withinthe scope of the present invention.

In accordance with embodiments of the present invention, the basisutilized to support a context model as described is any or a combinationof, but not necessarily limited to, the following:

-   -   1. Identity of the lexical items    -   2. Identity of surrounding lexical items (context)    -   3. Capitalization patterns    -   4. Parts of speech (not knowledge-poor)    -   5. Length of words    -   6. Word clustering    -   7. Punctuation    -   8. Syntax (not knowledge-poor)    -   9. Hand written rules (add a human layer)

In accordance with one embodiment, a basis for application of thecontext model is derived based on an algorithm and processing methoddisclosed in Yarowsky, D., Word-sense Disambibuation Using StatisticalModels of Roget's Categories Trained On Large Corpora, In Proc.COLING92, Nantes, France (1992). This reference discloses an algorithmthe primary intention of which is to disambiguate words that havesemantically different senses (river “bank” versus money “bank”). In oneembodiment, a similar algorithm is implemented to distinguish amongalternation patterns, thereby enabling a context model for applicationwithin the paraphrase system as described herein.

In accordance with block 256 in FIG. 3, it has been determined that if apattern is suitable to be applied to input 244, the next step is todetermine how specifically to apply the pattern. In other words, theboundaries of the text segment that should be modified are determined.Within a system that is not knowledge-poor, information such as parts ofspeech might support a hypothesis as to the location of boundaries;however, the present knowledge-poor method offers greater cross-domainand cross-linguistic applicability and generality. This being said,boundary detection methods that are not consistent with knowledge-poorprocessing are also within the scope of the present invention.

In accordance with one embodiment of the present invention, as isindicated by block 256 in FIG. 3, a language model 248 (FIG. 2) isimplemented to facilitate the process of applying an applicable patternto input 244. In one embodiment, language model 248 is trained on anindependent large corpus of data that is similar in nature to data 202(FIG. 2). A different data source, internal or otherwise could beutilized without departing from the scope of the present invention. Inaccordance with one embodiment, language model 249 is trained on newsdata (e.g., 3 years of the Wall Street Journal) when clusters 218 arenews-oriented.

In accordance with one aspect of the present invention, an applicablepattern is applied in multiple ways to input 244 in order to produce acandidate set of output text segments. Each candidate is compared tolanguage model 248 to determine the most likely candidate based onfrequency of occurrence. In this manner, a determination can be made asto how to best apply an applicable pattern to input 244. In accordancewith step 258 in FIG. 3, a paraphrase generator 249 receives applicablepatterns and appropriately processes input 244 with assistance fromcontext model 246 and language model 248 to produce a paraphrase 251.

An example will now be provided. An input text 244 is illustratively“President George Bush and Senator John Kerry”. A determination is madethat the patterns above the threshold include an applicable patternhaving the following form:A and B=>B and AA context model is then applied to facilitate a determination that theinput text 244 is similar enough to the data that motivated the patternto warrant applying the pattern to the input. Next, different textualvariations representing different applications of the pattern willdifferent textual boundaries are compared against a language model todetermine which variation is the most common. For example, differentnumbers of words are exchanged from either side of the word “and,” whichis a pivot point associated with the pattern. Accordingly, the followingpatterns are checked against the language model:

-   1. President George Kerry and Senator John Bush-   2. President John Bush and Senator George Kerry-   3. Senator George Bush and President John Kerry-   4. John George Bush and Senator President Kerry-   5. President George Senator and Bush John Kerry-   6. Kerry George Bush and Senator John President-   7. President George John and Senator Bush Kerry-   8. President Senator Bush and George John Kerry-   9. President Kerry Bush and Senator John George-   10. Senator John Bush and President George Kerry-   11. President John Kerry and Senator George Bush-   12. Senator John Kerry and President George Bush-   13. etc.    The comparison against the language model makes it evident that that    option #12 is most likely the best choice for application of the    pattern because that combination corresponds to the most frequent    combination of overlapping text segments (n-grams) as reflected in    the data. Finally, option #12 is provided as a meaning preserving    paraphrased adaptation of input 244.

It should be noted that any system for generating combinations forcomparison to the language model is within the scope of the presentinvention. The present invention is specifically not limited toexchanging words around a pivot point (e.g., a pivot word or phrase).

There are a wide variety of applications that will benefit from aparaphrase processing system as described herein. For example, thepotential applications for paraphrase recognition include a questionanswering system, as well as more general information retrieval systems.Additionally, such systems can use paraphrase generation capability fora variety of purposes including query expansion (e.g., to producemultiple forms of a single, original query) in order to find bettermatching results or to improve recall.

Still other applications for the described paraphrase functionalityinclude the summarization of multiple documents. By utilizing paraphraserecognition, an automatic document summarization system can find similarpassages in different documents to decide the most salient informationin the document set in order to generate a summary.

Another relevant application is a dialog system. Such a system cangenerate a response that echoes input, but is phrased differently toavoid parroting back the exact same input. This renders the dialogsystem more natural or conversational sounding.

The technology can also be used in word processing systems. The wordprocessing system can be configured to generate stylistic rewritesautomatically, and propose those rewrites to the user. This may behelpful, for instance, where a user is authoring a document and hasrepeated a phrase a large number of times, perhaps even in a singleparagraph. Similarly, a word processing system might include a featurethat flags repeated (but differently phrased) information that is spreadthroughout a document. Similarly, such a system may include a featurethat rewrites a piece of prose as a paraphrase.

Embodiments of the present invention can also be implemented in thecontext of command and control systems. People conventionally ask forthings using widely varying terminology. Identifying paraphrases allowssuch a system to implement the proper command and control actions evenif the inputs are phrased in varying ways, and may obviate the need fordetailed human scripting.

Although the present invention has been described with reference toparticular embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

1. A computer-readable storage medium having computer-executableinstructions that, when executed by a computing device, cause thecomputing device to perform a method comprising: identifying from acollection of data a plurality of patterns that represent paraphrasealternations, wherein each pattern is a unified string of text thatincludes an at least partially abstract representation of a first set ofwords in combination with an at least partially abstract representationof a corresponding second set of words, the representations being atleast partially abstract in that they include at least one placeholderrepresentation of a word rather than a specifically required word, andwherein the pattern of transformation indicates a pattern oftransformation from the first set of words to the second set of words,and wherein identifying comprises automatically identifying, and whereinidentifying is a step performed by a processing device that is afunctional component of the computing device; ranking the plurality ofpatterns based on frequency of occurrence within the collection of data;determining a threshold to be applied to the plurality of patterns, thethreshold being based on frequency of occurrence within the collectionof data; selecting a particular pattern from the plurality, theparticular pattern having a number of occurrences within the collectionof data, the number being greater than the determined threshold;applying the particular pattern to an input string so as to create aparaphrased version of the input string; and applying a language modelto determine how to apply the particular pattern to the input string,wherein applying a language model comprises applying a language modelthat incorporates data that reflects a subject matter that is alsoreflected in said collection of data.
 2. The computer-readable medium ofclaim 1, wherein identifying from a collection of data comprisesidentifying from a database of news articles.
 3. The computer-readablemedium of claim 1, wherein identifying from a collection of datacomprises identifying from a database of articles having related subjectmatter.
 4. The computer-readable medium of claim 1, wherein identifyingfrom a collection of data comprises identifying from a databasecontaining bilingual data.
 5. The computer-readable medium of claim 1,wherein identifying comprises clustering at least two related documentsand selecting at least one of said plurality of patterns based on acomparison of information contained in each document.
 6. Thecomputer-readable medium of claim 5, wherein identifying furthercomprises aligning related text segments.
 7. The computer-readablemedium of claim 1, further comprising generating a context model thatcorresponds to the particular pattern.
 8. The computer-readable mediumof claim 7, further comprising applying the context model to support adetermination as to whether to apply the particular pattern to the inputstring.
 9. The computer-readable medium of claim 8, further comprisingapplying a language model to determine how to apply the particularpattern to the input string.
 10. The computer-readable medium of claim9, wherein applying a language model comprises: applying the particularpattern to the input string in a plurality different ways to produce aplurality of different textual variations; and applying the languagemodel to the plurality of variations to determine a probable sequence ofwords.
 11. The computer-readable medium of claim 9, wherein applying alanguage model comprises applying a language model that incorporatesdata that reflects a subject matter that is also reflected in saidcollection of data.
 12. The computer-readable medium of claim 1, whereinapplying a language model comprises: applying the particular pattern tothe input string in a plurality different ways to produce a plurality ofdifferent textual variations; and applying the language model to theplurality of variations to determine probable sequence of words.
 13. Thecomputer-readable medium of claim 12, wherein identifying from acollection of data comprises identifying from a database of newsarticles.
 14. A computer implemented method of selectively applying aparaphrase alternation pattern to a textual input, comprising:identifying a plurality of potentially applicable paraphrase alternationpatterns, wherein each potentially applicable paraphrase alternationpattern is a unified string of text that includes an at least partiallyabstract representation of a first set of words in combination with anat least partially abstract representation of a corresponding second setof words, the representations being at least partially abstract in thatthey include at least one placeholder representation of a word ratherthan a specifically required word, and wherein the pattern oftransformation indicates a pattern of transformation from the first setof words to the second set of words, and wherein identifying comprisesautomatically identifying, and wherein identifying is a step performedby a processor that is a functional component of the computer;identifying from said plurality at least one applicable paraphrasealternation pattern; applying a context model to determine whether theapplicable paraphrase alternation pattern can be applied to the textualinput so as to preserve meaning; wherein the plurality of potentiallyapplicable paraphrase alternation patterns are limited to patterns thatoccur within a database enough times to rise above a predeterminedthreshold based on number of appearances within the database; andwherein identifying a plurality of potentially applicable paraphrasealternation patterns comprises clustering at least two related documentsand selecting at least one of said plurality of potentially applicableparaphrase alternation patterns based on a comparison of informationcontained in each document.
 15. The method of claim 14, whereinidentifying a plurality of potentially applicable paraphrase alternationpatterns comprises identifying from a database of articles havingrelated subject matter.
 16. The method of claim 14, wherein identifyinga plurality of potentially applicable paraphrase alternation patternscomprises identifying from a database containing bilingual data.
 17. Themethod of claim 14, wherein identifying further comprises aligningrelated text segments.
 18. The method of claim 17, wherein aligningrelated text segments comprises aligning on a string-identical basis.19. The method of claim 17, wherein aligning related text segmentscomprises aligning on a basis other than a string-identical rationale.20. The method of claim 14, applying a context model comprises applyinga context model generated based on data underlying the applicableparaphrase alternation pattern.
 21. The method of claim 14, furthercomprising applying a language model to determine how best to apply theapplicable paraphrase alternation pattern to the textual input.
 22. Themethod of claim 21, wherein applying a language model comprises:applying the applicable paraphrase alternation pattern to the textualinput in a plurality of different ways to produce a plurality ofdifferent textual variations; and applying the language model to theplurality of variations to determine probable sequence of words.
 23. Themethod of claim 22, wherein applying a language model comprises applyinga language model that incorporates data that reflects subject matterthat is also reflected in data from which the plurality of potentiallyapplicable paraphrase alternation patterns were derived.
 24. The methodof claim 14, wherein identifying comprises automatically identifying ona knowledge-poor basis.
 25. A computer-implemented system forselectively applying a paraphrase alternation pattern to a textualinput, comprising: a pattern identification component for processing acollection of data and automatically identifying a plurality ofpotentially applicable paraphrase transition patterns, wherein eachpotentially applicable paraphrase transition pattern is a unified stringof text that includes an at least partially abstract representation of afirst set of words in combination with an at least partially abstractrepresentation of a corresponding second set of words, therepresentations being at least partially abstract in that they includeat least one placeholder representation of a word rather than aspecifically required word, and wherein the pattern of transformationindicates a pattern of transformation from the first set of words to thesecond set of words, wherein said processing and automaticidentification are functions performed by a processor that is afunctional component of the computer; a context model that provides anobjective frame of reference in which to compare at least a first one ofthe plurality of potentially applicable paraphrase transition patternsto the textual input to determine whether the pattern can be applied soas to preserve meaning; and wherein the plurality of potentiallyapplicable paraphrase transition patterns is limited to a plurality ofpatterns that are determined to occur enough times in the collection ofdata to rise above a predetermined threshold based on number ofappearances in the collection of data.
 26. The system of claim 25,further comprising: a language model that provides an objective frame ofreference in which to determine how the first one of the plurality ofpotentially applicable paraphrase transition patterns best be applied tothe textual input.